error metric
Machine Learning Epidemic Predictions Using Agent-based Wireless Sensor Network Models
Nwokoye, Chukwunonso Henry, Oluchi, Blessing, Waldron, Sharna, Ezzeh, Peace
Given Name Surname line 2: dept. Abstract -- The lack of epidemiological data in wireless sensor networks (WSNs) is a fundamental difficulty in constructing robust models to forecast and mitigate threats like viruses and worms. Many studies have looked at different epidemic models for WSNs, focusing on the manner in which malware infections spread given the network's specific properties, including energy limits and node mobili ty. In this study, an agent - based realization of the susceptible - exposed - infected - recovered - vaccinated (SEIRV) mathematical model was employed for machine learning (ML) predictions. Using tools such as Netlogo's BehaviorSpace and Python, two epidemic synth etic datasets were generated and prepared for the application of several ML algorithms. Posed as a regression problem, the infected and recovered nodes were predicted, and the performance of these algorithms is compared using the error metrics of the train and the test sets. The predictions performed quite well, with low error metrics and high R values (0.997, 1.000, 0.999, 1.000), indicating an effective fit to the training set. The validation values were lowered (0.992, 0.998, 0.971, and 0.999), as is ty pical when evaluating model performance on unknown data. Judging from the recorded performances, support vector, linear, Lasso, Ridge, and ElasticNet regression were among the worst performing algorithms, while Random Forest, XGBoost, Decision Trees, and K nearest neighbor had the best model performances. In recent years, the globe as we know it has been changing due to bre akthroughs in numerous linked innovations including smart electrical grids [1], the IoT, long - term evolution, 5G connectivity [2] and cyber physical systems [3] such as wireless sensor networks (WSN).
Benchmarking Pre-Trained Time Series Models for Electricity Price Forecasting
Sartipi, Timothรฉe Hornek Amir, Tchappi, Igor, Fridgen, Gilbert
Accurate electricity price forecasting (EPF) is crucial for effective decision-making in power trading on the spot market. While recent advances in generative artificial intelligence (GenAI) and pre-trained large language models (LLMs) have inspired the development of numerous time series foundation models (TSFMs) for time series forecasting, their effectiveness in EPF remains uncertain. To address this gap, we benchmark several state-of-the-art pretrained models--Chronos-Bolt, Chronos-T5, TimesFM, Moirai, Time-MoE, and TimeGPT--against established statistical and machine learning (ML) methods for EPF. Using 2024 day-ahead auction (DAA) electricity prices from Germany, France, the Netherlands, Austria, and Belgium, we generate daily forecasts with a one-day horizon. Chronos-Bolt and Time-MoE emerge as the strongest among the TSFMs, performing on par with traditional models. However, the biseasonal MSTL model, which captures daily and weekly seasonality, stands out for its consistent performance across countries and evaluation metrics, with no TSFM statistically outperforming it.
A Perfectly Truthful Calibration Measure
Hartline, Jason, Hu, Lunjia, Wu, Yifan
Calibration requires that predictions are conditionally unbiased and, therefore, reliably interpretable as probabilities. Calibration measures quantify how far a predictor is from perfect calibration. As introduced by Haghtalab et al. (2024), a calibration measure is truthful if it is minimized in expectation when a predictor outputs the ground-truth probabilities. Although predicting the true probabilities guarantees perfect calibration, in reality, when calibration is evaluated on a finite sample, predicting the truth is not guaranteed to minimize any known calibration measure. All known calibration measures incentivize predictors to lie in order to appear more calibrated on a finite sample. Such lack of truthfulness motivated Haghtalab et al. (2024) and Qiao and Zhao (2025) to construct approximately truthful calibration measures in the sequential prediction setting, but no perfectly truthful calibration measure was known to exist even in the more basic batch setting. We design a perfectly truthful calibration measure in the batch setting: averaged two-bin calibration error (ATB). In addition to being truthful, ATB is sound, complete, continuous, and quadratically related to two existing calibration measures: the smooth calibration error (smCal) and the (lower) distance to calibration (distCal). The simplicity in our definition of ATB makes it efficient and straightforward to compute. ATB allows faster estimation algorithms with significantly easier implementations than smCal and distCal, achieving improved running time and simplicity for the calibration testing problem studied by Hu et al. (2024). We also introduce a general recipe for constructing truthful measures, which proves the truthfulness of ATB as a special case and allows us to construct other truthful calibration measures such as quantile-binned l_2-ECE.
NICE^k Metrics: Unified and Multidimensional Framework for Evaluating Deterministic Solar Forecasting Accuracy
Voyant, Cyril, Despotovic, Milan, Garcia-Gutierrez, Luis, Silva, Rodrigo Amaro e, Lauret, Philippe, Soubdhan, Ted, Bailek, Nadjem
Accurate solar energy output prediction is key for integrating renewables into grids, maintaining stability, and improving energy management. However, standard error metrics such as Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), and Skill Scores (SS) fail to capture the multidimensional nature of solar irradiance forecasting. These metrics lack sensitivity to forecastability, rely on arbitrary baselines (e.g., clear-sky models), and are poorly suited for operational use. To address this, we introduce the NICEk framework (Normalized Informed Comparison of Errors, with k = 1, 2, 3, Sigma), offering a robust and interpretable evaluation of forecasting models. Each NICEk score corresponds to an Lk norm: NICE1 targets average errors, NICE2 emphasizes large deviations, NICE3 highlights outliers, and NICESigma combines all. Using Monte Carlo simulations and data from 68 stations in the Spanish SIAR network, we evaluated methods including autoregressive models, extreme learning, and smart persistence. Theoretical and empirical results align when assumptions hold (e.g., R^2 ~ 1.0 for NICE2). Most importantly, NICESigma consistently shows higher discriminative power (p < 0.05), outperforming traditional metrics (p > 0.05). The NICEk metrics exhibit stronger statistical significance (e.g., p-values from 10^-6 to 0.004 across horizons) and greater generalizability. They offer a unified and operational alternative to standard error metrics in deterministic solar forecasting.
Using Deep Operators to Create Spatio-temporal Surrogates for Dynamical Systems under Uncertainty
Tang, Jichuan, Brewick, Patrick T., McClarren, Ryan G., Sweet, Christopher
Spatio-temporal data, which consists of responses or measurements gathered at different times and positions, is ubiquitous across diverse applications of civil infrastructure. While SciML methods have made significant progress in tackling the issue of response prediction for individual time histories, creating a full spatial-temporal surrogate remains a challenge. This study proposes a novel variant of deep operator networks (DeepONets), namely the full-field Extended DeepONet (FExD), to serve as a spatial-temporal surrogate that provides multi-output response predictions for dynamical systems. The proposed FExD surrogate model effectively learns the full solution operator across multiple degrees of freedom by enhancing the expressiveness of the branch network and expanding the predictive capabilities of the trunk network. The proposed FExD surrogate is deployed to simultaneously capture the dynamics at several sensing locations along a testbed model of a cable-stayed bridge subjected to stochastic ground motions. The ensuing response predictions from the FExD are comprehensively compared against both a vanilla DeepONet and a modified spatio-temporal Extended DeepONet. The results demonstrate the proposed FExD can achieve both superior accuracy and computational efficiency, representing a significant advancement in operator learning for structural dynamics applications.
Estimating properties of a homogeneous bounded soil using machine learning models
Kalimeris, Konstantinos, Mindrinos, Leonidas, Pallikarakis, Nikolaos
This work focuses on estimating soil properties from water moisture measurements. We consider simulated data generated by solving the initial-boundary value problem governing vertical infiltration in a homogeneous, bounded soil profile, with the usage of the Fokas method. To address the parameter identification problem, which is formulated as a two-output regression task, we explore various machine learning models. The performance of each model is assessed under different data conditions: full, noisy, and limited. Overall, the prediction of diffusivity $D$ tends to be more accurate than that of hydraulic conductivity $K.$ Among the models considered, Support Vector Machines (SVMs) and Neural Networks (NNs) demonstrate the highest robustness, achieving near-perfect accuracy and minimal errors.
Composable and adaptive design of machine learning interatomic potentials guided by Fisher-information analysis
Wang, Weishi, Transtrum, Mark K., Lordi, Vincenzo, Bulatov, Vasily V., Samanta, Amit
An adaptive physics-informed model design strategy for machine-learning interatomic potentials (MLIPs) is proposed. This strategy follows an iterative reconfiguration of composite models from single-term models, followed by a unified training procedure. A model evaluation method based on the Fisher information matrix (FIM) and multiple-property error metrics is proposed to guide model reconfiguration and hyperparameter optimization. Combining the model reconfiguration and the model evaluation subroutines, we provide an adaptive MLIP design strategy that balances flexibility and extensibility. In a case study of designing models against a structurally diverse niobium dataset, we managed to obtain an optimal configuration with 75 parameters generated by our framework that achieved a force RMSE of 0.172 eV/ร and an energy RMSE of 0.013 eV/atom.
Dynamical errors in machine learning forecasts
Fang, Zhou, Mengaldo, Gianmarco
In machine learning forecasting, standard error metrics such as mean absolute error (MAE) and mean squared error (MSE) quantify discrepancies between predictions and target values. However, these metrics do not directly evaluate the physical and/or dynamical consistency of forecasts, an increasingly critical concern in scientific and engineering applications. Indeed, a fundamental yet often overlooked question is whether machine learning forecasts preserve the dynamical behavior of the underlying system. Addressing this issue is essential for assessing the fidelity of machine learning models and identifying potential failure modes, particularly in applications where maintaining correct dynamical behavior is crucial. In this work, we investigate the relationship between standard forecasting error metrics, such as MAE and MSE, and the dynamical properties of the underlying system. To achieve this goal, we use two recently developed dynamical indices: the instantaneous dimension ($d$), and the inverse persistence ($ฮธ$). Our results indicate that larger forecast errors -- e.g., higher MSE -- tend to occur in states with higher $d$ (higher complexity) and higher $ฮธ$ (lower persistence). To further assess dynamical consistency, we propose error metrics based on the dynamical indices that measure the discrepancy of the forecasted $d$ and $ฮธ$ versus their correct values. Leveraging these dynamical indices-based metrics, we analyze direct and recursive forecasting strategies for three canonical datasets -- Lorenz, Kuramoto-Sivashinsky equation, and Kolmogorov flow -- as well as a real-world weather forecasting task. Our findings reveal substantial distortions in dynamical properties in ML forecasts, especially for long forecast lead times or long recursive simulations, providing complementary information on ML forecast fidelity that can be used to improve ML models.
Evaluating Large Language Models in Code Generation: INFINITE Methodology for Defining the Inference Index
Christakis, Nicholas, Drikakis, Dimitris
This study introduces a new methodology for an Inference Index (InI), called INFerence INdex In Testing model Effectiveness methodology (INFINITE), aiming to evaluate the performance of Large Language Models (LLMs) in code generation tasks. The InI index provides a comprehensive assessment focusing on three key components: efficiency, consistency, and accuracy. This approach encapsulates time-based efficiency, response quality, and the stability of model outputs, offering a thorough understanding of LLM performance beyond traditional accuracy metrics. We applied this methodology to compare OpenAI's GPT-4o (GPT), OpenAI-o1 pro (OAI1), and OpenAI-o3 mini-high (OAI3) in generating Python code for the Long-Short-Term-Memory (LSTM) model to forecast meteorological variables such as temperature, relative humidity and wind velocity. Our findings demonstrate that GPT outperforms OAI1 and performs comparably to OAI3 regarding accuracy and workflow efficiency. The study reveals that LLM-assisted code generation can produce results similar to expert-designed models with effective prompting and refinement. GPT's performance advantage highlights the benefits of widespread use and user feedback.
Model-free system identification of surface ships in waves via Hankel dynamic mode decomposition with control
Palma, Giorgio, Serani, Andrea, Aram, Shawn, Wundrow, David W., Drazen, David, Diez, Matteo
This study introduces and compares the Hankel dynamic mode decomposition with control (Hankel-DMDc) and a novel Bayesian extension of Hankel-DMDc as model-free (i.e., data-driven and equation-free) approaches for system identification and prediction of free-running ship motions in irregular waves. The proposed DMDc methods create a reduced-order model using limited data from the system state and incoming wave elevation histories, with the latter and rudder angle serving as forcing inputs. The inclusion of delayed states of the system as additional dimensions per the Hankel-DMDc improves the representation of the underlying non-linear dynamics of the system by DMD. The approaches are statistically assessed using data from free-running simulations of a 5415M hull's course-keeping in irregular beam-quartering waves at sea state 7, a highly severe condition characterized by nonlinear responses near roll-resonance. The results demonstrate robust performance and remarkable computational efficiency. The results indicate that the proposed methods effectively identify the dynamic system in analysis. Furthermore, the Bayesian formulation incorporates uncertainty quantification and enhances prediction accuracy. Ship motions are predicted with good agreement with test data over a 15 encounter waves observation window. No significant accuracy degradation is noted along the test sequences, suggesting the method can support accurate and efficient maritime design and operational planning.